Automated training and selection of models for document analysis

ABSTRACT

Embodiments are directed to a machine learning engine that determines training documents and validation documents from a plurality of documents. The machine learning engine may determine attributes associated with the documents. In response to receiving a request to predict attribute values of a selected document the machine learning engine may train a plurality of ML models to predict the attribute values based on the training documents and the attributes and associate the trained ML models with an accuracy score. The machine learning engine may determine candidate ML models from the trained ML models based on the training accuracy scores. The machine learning engine may evaluate and rank the candidate ML models based on the request and the validation documents. The machine learning engine may generate confirmed ML models based on the ranked candidate ML models such that the confirmed ML models may answer the request.

TECHNICAL FIELD

The present invention relates generally to document management, and moreparticularly, but not exclusively, to analyzing documents and contracts.

BACKGROUND

Modern organizations are often required to enter into complex orexpansive contracts between each other, vendors, suppliers, clients, orthe like. In some cases, complex contracts may require a significantamount of effort to establish between parties. This effort may includecommon activities, such as, term negotiating, back-and-forth review,local government approval, or the like. Likewise, some contracts havingparticular characteristics, such as, parties, subject matter, locale,terms, value/cost, or the like, may be more likely to result insuccessful outcomes than some other contracts having differentcharacteristics. For example, a routine service or production contractbetween two domestic organizations may have less risk than a contract toprovide a raw material from a remote area prone to local disruption(e.g., political upheaval, logistical problems, extreme weather events,or the like). Often, organizations may have little insight into how themany characteristics of a complex contract, such as, parties, politicalconditions, choice of law, venues, forums, geographic locale, subjectmatter, value/cost, or the like, may impact the process of obtaining anexecuted contract as well as how contract characteristics impact thelikelihood of contract performance. Accordingly, in some cases, it maybe difficult for organizations to predict the time or effort it may taketo obtain a particular contract. Likewise, it may be difficult fororganizations to predict the potential of non-performance for mitigationplanning. Thus, it is with respect to these considerations and othersthat the present invention has been made.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present innovationsare described with reference to the following drawings. In the drawings,like reference numerals refer to like parts throughout the variousfigures unless otherwise specified. For a better understanding of thedescribed innovations, reference will be made to the following DetailedDescription of Various Embodiments, which is to be read in associationwith the accompanying drawings, wherein:

FIG. 1 illustrates a system environment in which various embodiments maybe implemented;

FIG. 2 illustrates a schematic embodiment of a client computer;

FIG. 3 illustrates a schematic embodiment of a network computer;

FIG. 4 illustrates a logical schematic of a system for automatedtraining and selection of models for document analysis in accordancewith one or more of the various embodiments;

FIG. 5 illustrates a logical schematic of a system for automatedtraining and select of models for document or contract analysis inaccordance with one or more of the various embodiments;

FIG. 6 illustrates an overview flowchart of a process for automatedtraining and select of models for document or contract analysis inaccordance with one or more of the various embodiments;

FIG. 7 illustrates a flowchart of a process for determining documentattributes for automated training and select of models for document orcontract analysis in accordance with one or more of the variousembodiments;

FIG. 8 illustrates a flowchart of a process for predicting unknowndocument attributes of a working document in accordance with one or moreof the various embodiments; and

FIG. 9 illustrates a flowchart of a process for training machinelearning models used for automated training and select of models fordocument or contract analysis in accordance with one or more of thevarious embodiments.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Various embodiments now will be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific exemplary embodiments bywhich the invention may be practiced. The embodiments may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the embodiments to those skilled in the art.Among other things, the various embodiments may be methods, systems,media or devices. Accordingly, the various embodiments may take the formof an entirely hardware embodiment, an entirely software embodiment oran embodiment combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments may be readily combined, withoutdeparting from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

For example embodiments, the following terms are also used hereinaccording to the corresponding meaning, unless the context clearlydictates otherwise.

As used herein the term, “engine” refers to logic embodied in hardwareor software instructions, which can be written in a programminglanguage, such as C, C++, Objective-C, COBOL, Java™, Kotlin, PHP, Perl,JavaScript, Ruby, VBScript, Microsoft .NET™ languages such as C#, or thelike. An engine may be compiled into executable programs or written ininterpreted programming languages. Software engines may be callable fromother engines or from themselves. Engines described herein refer to oneor more logical modules that can be merged with other engines orapplications, or can be divided into sub-engines. The engines can bestored in non-transitory computer-readable medium or computer storagedevice and be stored on and executed by one or more general purposecomputers, thus creating a special purpose computer configured toprovide the engine. Also, in some embodiments, one or more portions ofan engine may be a hardware device, ASIC, FPGA, or the like, thatperforms one or more actions in the support of an engine or as part ofthe engine.

As used herein the term, “evaluator” refers to a package or bundle ofcomputer readable instructions, configuration information, rules,patterns, regular expressions, condition logic, branching logic,software libraries, FPGAs, ASICs, or the like, or combination thereofthat may be used to evaluate documents or document clauses. In somecases, evaluators may be used determine characteristics about a contractincluding one or more attributes or features of the contract. Variousevaluators may be specialized for identifying or validating one or morecategories of clauses or validating one or more document or contracttypes. In some embodiments, organizations or users may provide customevaluators to identify clause categories or document types that may beunique to their organization.

The following briefly describes embodiments of the invention in order toprovide a basic understanding of some aspects of the invention. Thisbrief description is not intended as an extensive overview. It is notintended to identify key or critical elements, or to delineate orotherwise narrow the scope. Its purpose is merely to present someconcepts in a simplified form as a prelude to the more detaileddescription that is presented later.

Briefly stated, various embodiments are directed to managing documentsover a network. In one or more of the various embodiments, a machinelearning engine may be instantiated to determine one or more of trainingdocuments and one or more validation documents that are randomlyselected from a plurality of documents.

In one or more of the various embodiments, the machine learning enginemay be arranged to determine one or more attributes that may beassociated with the plurality of documents such that the one or moreattributes may be associated with one or more features that may beassociated with the plurality of documents.

In one or more of the various embodiments, the machine learning enginemay be arranged to determine one or more categorical attributes thathave values that are names or labels; and determine one or morenumerical attributes that may have values that represent a numericalmeaning. In some embodiments, the one or more attributes may include oneor more of value, number of cycle day, geographic location, subjectmatter, duration, delivery date, entities, forum, venue, or the like.

In one or more of the various embodiments, the machine learning enginemay be arranged to perform further actions in response to receiving arequest to predict one or more attribute values of a selected document.

In one or more of the various embodiments, the machine learning enginemay be arranged to train a plurality of ML models to predict the one ormore attribute values based on the one or more training documents andthe one or more attributes such that each trained ML model may beassociated with a training accuracy score. In one or more of the variousembodiments, one or more of the plurality of ML models may be eitherfully trained or partially trained prior to receiving the request.

In one or more of the various embodiments, the machine learning enginemay be arranged to determine one or more candidate ML models from theplurality of trained ML models based on each associated trainingaccuracy score exceeding a threshold value.

In one or more of the various embodiments, the machine learning enginemay be arranged to evaluate the one or more candidate ML models based onthe request and the one or more validation documents such that each ofthe one or more evaluated candidate ML models may be ranked. In one ormore of the various embodiments, determining the one or more candidateML models may include, in response to the request matching one or moreprevious requests, modifying the set of one or more candidate ML modelsto include one or more confirmed ML models that were previously used foranswering the one or more matched requests.

In one or more of the various embodiments, the machine learning enginemay be arranged to generate one or more confirmed ML models based on theone or more ranked candidate ML models such that the one or moreconfirmed ML models may be employed to answer the request and predictthe one or more attribute values of the selected document, and such thatemploying the one or more confirmed ML models improves both efficiencyof employed computing resources and accuracy of the answer to therequest.

In one or more of the various embodiments, the machine learning enginemay be arranged to provide one or more document types based on aselection of attributes or attribute values such that each document maybe associated with the one or more document types based on the one ormore attributes included in the document; and, in some embodiments,associating one or more of the plurality of ML models with the one ormore document types.

In one or more of the various embodiments, the machine learning enginemay be arranged to execute the one or more confirmed ML models toidentify which of the one or more attributes of the document may beoutliers.

In one or more of the various embodiments, the machine learning enginemay be arranged to execute the one or more confirmed ML models toidentify one or more document clusters based on the one or moreattributes such that the plurality of documents are associated with theone or more clusters based on values of the one or more attributes thatare associated with each cluster of documents.

Illustrated Operating Environment

FIG. 1 shows components of one embodiment of an environment in whichembodiments of the invention may be practiced. Not all of the componentsmay be required to practice the invention, and variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the invention. As shown, system 100 of FIG.1 includes local area networks (LANs)/wide area networks(WANs)—(network) 110, wireless network 108, client computers 102-105,document analysis server computer 116, document management servercomputer 118, or the like.

At least one embodiment of client computers 102-105 is described in moredetail below in conjunction with FIG. 2. In one embodiment, at leastsome of client computers 102-105 may operate over one or more wired orwireless networks, such as networks 108, or 110. Generally, clientcomputers 102-105 may include virtually any computer capable ofcommunicating over a network to send and receive information, performvarious online activities, offline actions, or the like. In oneembodiment, one or more of client computers 102-105 may be configured tooperate within a business or other entity to perform a variety ofservices for the business or other entity. For example, client computers102-105 may be configured to operate as a web server, firewall, clientapplication, media player, mobile telephone, game console, desktopcomputer, or the like. However, client computers 102-105 are notconstrained to these services and may also be employed, for example, asfor end-user computing in other embodiments. It should be recognizedthat more or less client computers (as shown in FIG. 1) may be includedwithin a system such as described herein, and embodiments are thereforenot constrained by the number or type of client computers employed.

Computers that may operate as client computer 102 may include computersthat typically connect using a wired or wireless communications mediumsuch as personal computers, multiprocessor systems, microprocessor-basedor programmable electronic devices, network PCs, or the like. In someembodiments, client computers 102-105 may include virtually any portablecomputer capable of connecting to another computer and receivinginformation such as, laptop computer 103, mobile computer 104, tabletcomputers 105, or the like. However, portable computers are not solimited and may also include other portable computers such as cellulartelephones, display pagers, radio frequency (RF) devices, infrared (IR)devices, Personal Digital Assistants (PDAs), handheld computers,wearable computers, integrated devices combining one or more of thepreceding computers, or the like. As such, client computers 102-105typically range widely in terms of capabilities and features. Moreover,client computers 102-105 may access various computing applications,including a browser, or other web-based application.

A web-enabled client computer may include a browser application that isconfigured to send requests and receive responses over the web. Thebrowser application may be configured to receive and display graphics,text, multimedia, and the like, employing virtually any web-basedlanguage. In one embodiment, the browser application is enabled toemploy JavaScript, HyperText Markup Language (HTML), eXtensible MarkupLanguage (XML), JavaScript Object Notation (JSON), Cascading StyleSheets (CSS), or the like, or combination thereof, to display and send amessage. In one embodiment, a user of the client computer may employ thebrowser application to perform various activities over a network(online). However, another application may also be used to performvarious online activities.

Client computers 102-105 also may include at least one other clientapplication that is configured to receive or send content betweenanother computer. The client application may include a capability tosend or receive content, or the like. The client application may furtherprovide information that identifies itself, including a type,capability, name, and the like. In one embodiment, client computers102-105 may uniquely identify themselves through any of a variety ofmechanisms, including an Internet Protocol (IP) address, a phone number,Mobile Identification Number (MIN), an electronic serial number (ESN), aclient certificate, or other device identifier. Such information may beprovided in one or more network packets, or the like, sent between otherclient computers, document analysis server computer 116, documentmanagement server computer 118, or other computers.

Client computers 102-105 may further be configured to include a clientapplication that enables an end-user to log into an end-user accountthat may be managed by another computer, such as document analysisserver computer 116, document management server computer 118, or thelike. Such an end-user account, in one non-limiting example, may beconfigured to enable the end-user to manage one or more onlineactivities, including in one non-limiting example, project management,software development, system administration, configuration management,search activities, social networking activities, browse variouswebsites, communicate with other users, or the like. Also, clientcomputers may be arranged to enable users to display reports,interactive user-interfaces, or results provided by document analysisserver computer 116.

Wireless network 108 is configured to couple client computers 103-105and its components with network 110. Wireless network 108 may includeany of a variety of wireless sub-networks that may further overlaystand-alone ad-hoc networks, and the like, to provide aninfrastructure-oriented connection for client computers 103-105. Suchsub-networks may include mesh networks, Wireless LAN (WLAN) networks,cellular networks, and the like. In one embodiment, the system mayinclude more than one wireless network.

Wireless network 108 may further include an autonomous system ofterminals, gateways, routers, and the like connected by wireless radiolinks, and the like. These connectors may be configured to move freelyand randomly and organize themselves arbitrarily, such that the topologyof wireless network 108 may change rapidly.

Wireless network 108 may further employ a plurality of accesstechnologies including 2nd (2G), 3rd (3G), 4th (4G) 5th (5G) generationradio access for cellular systems, WLAN, Wireless Router (WR) mesh, andthe like. Access technologies such as 2G, 3G, 4G, 5G, and future accessnetworks may enable wide area coverage for mobile computers, such asclient computers 103-105 with various degrees of mobility. In onenon-limiting example, wireless network 108 may enable a radio connectionthrough a radio network access such as Global System for Mobilcommunication (GSM), General Packet Radio Services (GPRS), Enhanced DataGSM Environment (EDGE), code division multiple access (CDMA), timedivision multiple access (TDMA), Wideband Code Division Multiple Access(WCDMA), High Speed Downlink Packet Access (HSDPA), Long Term Evolution(LTE), and the like. In essence, wireless network 108 may includevirtually any wireless communication mechanism by which information maytravel between client computers 103-105 and another computer, network, acloud-based network, a cloud instance, or the like.

Network 110 is configured to couple network computers with othercomputers, including, document analysis server computer 116, documentmanagement server computer 118, client computers 102, and clientcomputers 103-105 through wireless network 108, or the like. Network 110is enabled to employ any form of computer readable media forcommunicating information from one electronic device to another. Also,network 110 can include the Internet in addition to local area networks(LANs), wide area networks (WANs), direct connections, such as through auniversal serial bus (USB) port, Ethernet port, other forms ofcomputer-readable media, or any combination thereof. On aninterconnected set of LANs, including those based on differingarchitectures and protocols, a router acts as a link between LANs,enabling messages to be sent from one to another. In addition,communication links within LANs typically include twisted wire pair orcoaxial cable, while communication links between networks may utilizeanalog telephone lines, full or fractional dedicated digital linesincluding T1, T2, T3, and T4, or other carrier mechanisms including, forexample, E-carriers, Integrated Services Digital Networks (ISDNs),Digital Subscriber Lines (DSLs), wireless links including satellitelinks, or other communications links known to those skilled in the art.Moreover, communication links may further employ any of a variety ofdigital signaling technologies, including without limit, for example,DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, or the like.Furthermore, remote computers and other related electronic devices couldbe remotely connected to either LANs or WANs via a modem and temporarytelephone link. In one embodiment, network 110 may be configured totransport information of an Internet Protocol (IP).

Additionally, communication media typically embodies computer readableinstructions, data structures, program modules, or other transportmechanism and includes any information non-transitory delivery media ortransitory delivery media. By way of example, communication mediaincludes wired media such as twisted pair, coaxial cable, fiber optics,wave guides, and other wired media and wireless media such as acoustic,RF, infrared, and other wireless media.

Also, one embodiment of document analysis server computer 116, documentmanagement server computer 118 are described in more detail below inconjunction with FIG. 3. Although FIG. 1 illustrates document analysisserver computer 116, document management server computer 118 each as asingle computer, the innovations or embodiments are not so limited. Forexample, one or more functions of document analysis server computer 116,document management server computer 118, or the like, may be distributedacross one or more distinct network computers. Moreover, in one or moreembodiments, document analysis server computer 116, document managementserver computer 118 may be implemented using a plurality of networkcomputers. Further, in one or more of the various embodiments, documentanalysis server computer 116, document management server computer 118,or the like, may be implemented using one or more cloud instances in oneor more cloud networks. Accordingly, these innovations and embodimentsare not to be construed as being limited to a single environment, andother configurations, and other architectures are also envisaged.

Illustrative Client computer

FIG. 2 shows one embodiment of client computer 200 that may include manymore or less components than those shown. Client computer 200 mayrepresent, for example, one or more embodiment of mobile computers orclient computers shown in FIG. 1.

Client computer 200 may include processor 202 in communication withmemory 204 via bus 228. Client computer 200 may also include powersupply 230, network interface 232, audio interface 256, display 250,keypad 252, illuminator 254, video interface 242, input/output interface238, haptic interface 264, global positioning systems (GPS) receiver258, open air gesture interface 260, temperature interface 262,camera(s) 240, projector 246, pointing device interface 266,processor-readable stationary storage device 234, and processor-readableremovable storage device 236. Client computer 200 may optionallycommunicate with a base station (not shown), or directly with anothercomputer. And in one embodiment, although not shown, a gyroscope may beemployed within client computer 200 to measuring or maintaining anorientation of client computer 200.

Power supply 230 may provide power to client computer 200. Arechargeable or non-rechargeable battery may be used to provide power.The power may also be provided by an external power source, such as anAC adapter or a powered docking cradle that supplements or recharges thebattery.

Network interface 232 includes circuitry for coupling client computer200 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,protocols and technologies that implement any portion of the OSI modelfor mobile communication (GSM), CDMA, time division multiple access(TDMA), UDP, TCP/IP, SMS, MMS, GPRS, WAP, UWB, WiMax, SIP/RTP, GPRS,EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or any of avariety of other wireless communication protocols. Network interface 232is sometimes known as a transceiver, transceiving device, or networkinterface card (MC).

Audio interface 256 may be arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 256 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others or generate an audio acknowledgement forsome action. A microphone in audio interface 256 can also be used forinput to or control of client computer 200, e.g., using voicerecognition, detecting touch based on sound, and the like.

Display 250 may be a liquid crystal display (LCD), gas plasma,electronic ink, light emitting diode (LED), Organic LED (OLED) or anyother type of light reflective or light transmissive display that can beused with a computer. Display 250 may also include a touch interface 244arranged to receive input from an object such as a stylus or a digitfrom a human hand, and may use resistive, capacitive, surface acousticwave (SAW), infrared, radar, or other technologies to sense touch orgestures.

Projector 246 may be a remote handheld projector or an integratedprojector that is capable of projecting an image on a remote wall or anyother reflective object such as a remote screen.

Video interface 242 may be arranged to capture video images, such as astill photo, a video segment, an infrared video, or the like. Forexample, video interface 242 may be coupled to a digital video camera, aweb-camera, or the like. Video interface 242 may comprise a lens, animage sensor, and other electronics. Image sensors may include acomplementary metal-oxide-semiconductor (CMOS) integrated circuit,charge-coupled device (CCD), or any other integrated circuit for sensinglight.

Keypad 252 may comprise any input device arranged to receive input froma user. For example, keypad 252 may include a push button numeric dial,or a keyboard. Keypad 252 may also include command buttons that areassociated with selecting and sending images.

Illuminator 254 may provide a status indication or provide light.Illuminator 254 may remain active for specific periods of time or inresponse to event messages. For example, when illuminator 254 is active,it may backlight the buttons on keypad 252 and stay on while the clientcomputer is powered. Also, illuminator 254 may backlight these buttonsin various patterns when particular actions are performed, such asdialing another client computer. Illuminator 254 may also cause lightsources positioned within a transparent or translucent case of theclient computer to illuminate in response to actions.

Further, client computer 200 may also comprise hardware security module(HSM) 268 for providing additional tamper resistant safeguards forgenerating, storing or using security/cryptographic information such as,keys, digital certificates, passwords, passphrases, two-factorauthentication information, or the like. In some embodiments, hardwaresecurity module may be employed to support one or more standard publickey infrastructures (PKI), and may be employed to generate, manage, orstore keys pairs, or the like. In some embodiments, HSM 268 may be astand-alone computer, in other cases, HSM 268 may be arranged as ahardware card that may be added to a client computer.

Client computer 200 may also comprise input/output interface 238 forcommunicating with external peripheral devices or other computers suchas other client computers and network computers. The peripheral devicesmay include an audio headset, virtual reality headsets, display screenglasses, remote speaker system, remote speaker and microphone system,and the like. Input/output interface 238 can utilize one or moretechnologies, such as Universal Serial Bus (USB), Infrared, WiFi, WiMax,Bluetooth™, and the like.

Input/output interface 238 may also include one or more sensors fordetermining geolocation information (e.g., GPS), monitoring electricalpower conditions (e.g., voltage sensors, current sensors, frequencysensors, and so on), monitoring weather (e.g., thermostats, barometers,anemometers, humidity detectors, precipitation scales, or the like), orthe like. Sensors may be one or more hardware sensors that collect ormeasure data that is external to client computer 200.

Haptic interface 264 may be arranged to provide tactile feedback to auser of the client computer. For example, the haptic interface 264 maybe employed to vibrate client computer 200 in a particular way whenanother user of a computer is calling. Temperature interface 262 may beused to provide a temperature measurement input or a temperaturechanging output to a user of client computer 200. Open air gestureinterface 260 may sense physical gestures of a user of client computer200, for example, by using single or stereo video cameras, radar, agyroscopic sensor inside a computer held or worn by the user, or thelike. Camera 240 may be used to track physical eye movements of a userof client computer 200.

GPS transceiver 258 can determine the physical coordinates of clientcomputer 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 258 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of client computer 200 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 258 can determine a physical location for clientcomputer 200. In one or more embodiment, however, client computer 200may, through other components, provide other information that may beemployed to determine a physical location of the client computer,including for example, a Media Access Control (MAC) address, IP address,and the like.

In at least one of the various embodiments, applications, such as,operating system 206, other client apps 224, web browser 226, or thelike, may be arranged to employ geo-location information to select oneor more localization features, such as, time zones, languages,currencies, calendar formatting, or the like. Localization features maybe used in documents, clauses, evaluators, machine learning models,user-interfaces, reports, as well as internal processes or databases. Inat least one of the various embodiments, geo-location information usedfor selecting localization information may be provided by GPS 258. Also,in some embodiments, geolocation information may include informationprovided using one or more geolocation protocols over the networks, suchas, wireless network 108 or network 111.

Human interface components can be peripheral devices that are physicallyseparate from client computer 200, allowing for remote input or outputto client computer 200. For example, information routed as describedhere through human interface components such as display 250 or keyboard252 can instead be routed through network interface 232 to appropriatehuman interface components located remotely. Examples of human interfaceperipheral components that may be remote include, but are not limitedto, audio devices, pointing devices, keypads, displays, cameras,projectors, and the like. These peripheral components may communicateover a Pico Network such as Bluetooth™, Zigbee™ and the like. Onenon-limiting example of a client computer with such peripheral humaninterface components is a wearable computer, which might include aremote pico projector along with one or more cameras that remotelycommunicate with a separately located client computer to sense a user'sgestures toward portions of an image projected by the pico projectoronto a reflected surface such as a wall or the user's hand.

A client computer may include web browser application 226 that isconfigured to receive and to send web pages, web-based messages,graphics, text, multimedia, and the like. The client computer's browserapplication may employ virtually any programming language, including awireless application protocol messages (WAP), and the like. In one ormore embodiment, the browser application is enabled to employ HandheldDevice Markup Language (HDML), Wireless Markup Language (WML),WMLScript, JavaScript, Standard Generalized Markup Language (SGML),HyperText Markup Language (HTML), eXtensible Markup Language (XML),HTML5, and the like.

Memory 204 may include RAM, ROM, or other types of memory. Memory 204illustrates an example of computer-readable storage media (devices) forstorage of information such as computer-readable instructions, datastructures, program modules or other data. Memory 204 may store BIOS 208for controlling low-level operation of client computer 200. The memorymay also store operating system 206 for controlling the operation ofclient computer 200. It will be appreciated that this component mayinclude a general-purpose operating system such as a version of UNIX, orLINUX™, or a specialized client computer communication operating systemsuch as Windows Phone™, or the Symbian® operating system. The operatingsystem may include, or interface with a Java virtual machine module thatenables control of hardware components or operating system operationsvia Java application programs.

Memory 204 may further include one or more data storage 210, which canbe utilized by client computer 200 to store, among other things,applications 220 or other data. For example, data storage 210 may alsobe employed to store information that describes various capabilities ofclient computer 200. The information may then be provided to anotherdevice or computer based on any of a variety of methods, including beingsent as part of a header during a communication, sent upon request, orthe like. Data storage 210 may also be employed to store socialnetworking information including address books, buddy lists, aliases,user profile information, or the like. Data storage 210 may furtherinclude program code, data, algorithms, and the like, for use by aprocessor, such as processor 202 to execute and perform actions. In oneembodiment, at least some of data storage 210 might also be stored onanother component of client computer 200, including, but not limited to,non-transitory processor-readable removable storage device 236,processor-readable stationary storage device 234, or even external tothe client computer.

Applications 220 may include computer executable instructions which,when executed by client computer 200, transmit, receive, or otherwiseprocess instructions and data. Applications 220 may include, forexample, other client applications 224, web browser 226, or the like.Client computers may be arranged to exchange communications, such as,document management operations, document administration, documentevaluation, document clause discovery, queries, searches, messages,notification messages, event messages, alerts, performance metrics, logdata, API calls, or the like, combination thereof, with documentanalysis server computers or document management server computers.

Other examples of application programs include calendars, searchprograms, email client applications, IM applications, SMS applications,Voice Over Internet Protocol (VOIP) applications, contact managers, taskmanagers, transcoders, database programs, word processing programs,security applications, spreadsheet programs, games, search programs, andso forth.

Additionally, in one or more embodiments (not shown in the figures),client computer 200 may include an embedded logic hardware deviceinstead of a CPU, such as, an Application Specific Integrated Circuit(ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic(PAL), or the like, or combination thereof. The embedded logic hardwaredevice may directly execute its embedded logic to perform actions. Also,in one or more embodiments (not shown in the figures), client computer200 may include one or more hardware microcontrollers instead of CPUs.In one or more embodiment, the one or more microcontrollers may directlyexecute their own embedded logic to perform actions and access its owninternal memory and its own external Input and Output Interfaces (e.g.,hardware pins or wireless transceivers) to perform actions, such asSystem On a Chip (SOC), or the like.

Illustrative Network Computer

FIG. 3 shows one embodiment of network computer 300 that may be includedin a system implementing one or more of the various embodiments. Networkcomputer 300 may include many more or less components than those shownin FIG. 3. However, the components shown are sufficient to disclose anillustrative embodiment for practicing these innovations. Networkcomputer 300 may represent, for example, one embodiment of at least oneof document analysis server computer 116, or document management servercomputer 118 of FIG. 1.

Network computers, such as, network computer 300 may include a processor302 that may be in communication with a memory 304 via a bus 328. Insome embodiments, processor 302 may be comprised of one or more hardwareprocessors, or one or more processor cores. In some cases, one or moreof the one or more processors may be specialized processors designed toperform one or more specialized actions, such as, those describedherein. Network computer 300 also includes a power supply 330, networkinterface 332, audio interface 356, display 350, keyboard 352,input/output interface 338, processor-readable stationary storage device334, and processor-readable removable storage device 336. Power supply330 provides power to network computer 300.

Network interface 332 includes circuitry for coupling network computer300 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,protocols and technologies that implement any portion of the OpenSystems Interconnection model (OSI model), global system for mobilecommunication (GSM), code division multiple access (CDMA), time divisionmultiple access (TDMA), user datagram protocol (UDP), transmissioncontrol protocol/Internet protocol (TCP/IP), Short Message Service(SMS), Multimedia Messaging Service (MMS), general packet radio service(GPRS), WAP, ultra-wide band (UWB), IEEE 802.16 WorldwideInteroperability for Microwave Access (WiMax), Session InitiationProtocol/Real-time Transport Protocol (SIP/RTP), or any of a variety ofother wired and wireless communication protocols. Network interface 332is sometimes known as a transceiver, transceiving device, or networkinterface card (NIC). Network computer 300 may optionally communicatewith a base station (not shown), or directly with another computer.

Audio interface 356 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 356 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others or generate an audio acknowledgement forsome action. A microphone in audio interface 356 can also be used forinput to or control of network computer 300, for example, using voicerecognition.

Display 350 may be a liquid crystal display (LCD), gas plasma,electronic ink, light emitting diode (LED), Organic LED (OLED) or anyother type of light reflective or light transmissive display that can beused with a computer. In some embodiments, display 350 may be a handheldprojector or pico projector capable of projecting an image on a wall orother object.

Network computer 300 may also comprise input/output interface 338 forcommunicating with external devices or computers not shown in FIG. 3.Input/output interface 338 can utilize one or more wired or wirelesscommunication technologies, such as USB™, Firewire™, WiFi, WiMax,Thunderbolt™, Infrared, Bluetooth™, Zigbee™, serial port, parallel port,and the like.

Also, input/output interface 338 may also include one or more sensorsfor determining geolocation information (e.g., GPS), monitoringelectrical power conditions (e.g., voltage sensors, current sensors,frequency sensors, and so on), monitoring weather (e.g., thermostats,barometers, anemometers, humidity detectors, precipitation scales, orthe like), or the like. Sensors may be one or more hardware sensors thatcollect or measure data that is external to network computer 300. Humaninterface components can be physically separate from network computer300, allowing for remote input or output to network computer 300. Forexample, information routed as described here through human interfacecomponents such as display 350 or keyboard 352 can instead be routedthrough the network interface 332 to appropriate human interfacecomponents located elsewhere on the network. Human interface componentsinclude any component that allows the computer to take input from, orsend output to, a human user of a computer. Accordingly, pointingdevices such as mice, styluses, track balls, or the like, maycommunicate through pointing device interface 358 to receive user input.

GPS transceiver 340 can determine the physical coordinates of networkcomputer 300 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 340 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of network computer 300 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 340 can determine a physical location for networkcomputer 300. In one or more embodiments, however, network computer 300may, through other components, provide other information that may beemployed to determine a physical location of the client computer,including for example, a Media Access Control (MAC) address, IP address,and the like.

In at least one of the various embodiments, applications, such as,operating system 306, document engine 322, insight engine 324, machinelearning engine 326, web services 329, or the like, may be arranged toemploy geo-location information to select one or more localizationfeatures, such as, time zones, languages, currencies, currencyformatting, calendar formatting, or the like. Localization features maybe used in documents, clauses, clause meta-data, file systems,user-interfaces, reports, textual evaluators, semantic evaluators, aswell as internal processes or databases. In at least one of the variousembodiments, geo-location information used for selecting localizationinformation may be provided by GPS 340. Also, in some embodiments,geolocation information may include information provided using one ormore geolocation protocols over the networks, such as, wireless network108 or network 111.

Memory 304 may include Random Access Memory (RAM), Read-Only Memory(ROM), or other types of memory. Memory 304 illustrates an example ofcomputer-readable storage media (devices) for storage of informationsuch as computer-readable instructions, data structures, program modulesor other data. Memory 304 stores a basic input/output system (BIOS) 308for controlling low-level operation of network computer 300. The memoryalso stores an operating system 306 for controlling the operation ofnetwork computer 300. It will be appreciated that this component mayinclude a general-purpose operating system such as a version of UNIX, orLINUX™, or a specialized operating system such as MicrosoftCorporation's Windows® operating system, or the Apple Corporation's OSX®operating system. The operating system may include, or interface withone or more virtual machine modules, such as, a Java virtual machinemodule that enables control of hardware components or operating systemoperations via Java application programs. Likewise, other runtimeenvironments may be included.

Memory 304 may further include one or more data storage 310, which canbe utilized by network computer 300 to store, among other things,applications 320 or other data. For example, data storage 310 may alsobe employed to store information that describes various capabilities ofnetwork computer 300. The information may then be provided to anotherdevice or computer based on any of a variety of methods, including beingsent as part of a header during a communication, sent upon request, orthe like. Data storage 310 may also be employed to store socialnetworking information including address books, buddy lists, aliases,user profile information, or the like. Data storage 310 may furtherinclude program code, data, algorithms, and the like, for use by aprocessor, such as processor 302 to execute and perform actions such asthose actions described below. In one embodiment, at least some of datastorage 310 might also be stored on another component of networkcomputer 300, including, but not limited to, non-transitory media insideprocessor-readable removable storage device 336, processor-readablestationary storage device 334, or any other computer-readable storagedevice within network computer 300, or even external to network computer300. Data storage 310 may include, for example, documents 314, machinelearning models 318, or the like. Documents 314 may store files,documents, versions, properties, meta-data, data structures, or thelike, that represent one or more portions of a document, including rawdocuments or documents that have undergone additional analysis such asclause discovery or insight analysis. Machine learning models 318 maystore one or more machine learning models that may be trained forautomated training and selection of models for document analysis.Applications 320 may include computer executable instructions which,when executed by network computer 300, transmit, receive, or otherwiseprocess messages (e.g., SMS, Multimedia Messaging Service (MMS), InstantMessage (IM), email, or other messages), audio, video, and enabletelecommunication with another user of another mobile computer. Otherexamples of application programs include calendars, search programs,email client applications, IM applications, SMS applications, Voice OverInternet Protocol (VOIP) applications, contact managers, task managers,transcoders, database programs, word processing programs, securityapplications, spreadsheet programs, games, search programs, and soforth. Applications 320 may include document engine 322, insight engine324, machine learning engine 326, web services 329, or the like, thatmay be arranged to perform actions for embodiments described below. Inone or more of the various embodiments, one or more of the applicationsmay be implemented as modules or components of another application.Further, in one or more of the various embodiments, applications may beimplemented as operating system extensions, modules, plugins, or thelike.

Furthermore, in one or more of the various embodiments, document engine322, insight engine 324, machine learning engine 326, web services 329,or the like, may be operative in a cloud-based computing environment. Inone or more of the various embodiments, these applications, and others,that comprise the management platform may be executing within virtualmachines or virtual servers that may be managed in a cloud-based basedcomputing environment. In one or more of the various embodiments, inthis context the applications may flow from one physical networkcomputer within the cloud-based environment to another depending onperformance and scaling considerations automatically managed by thecloud computing environment. Likewise, in one or more of the variousembodiments, virtual machines or virtual servers dedicated to documentengine 322, insight engine 324, machine learning engine 326, webservices 329, or the like, may be provisioned and de-commissionedautomatically.

Also, in one or more of the various embodiments, document engine 322,insight engine 324, machine learning engine 326, web services 329, orthe like, may be located in virtual servers running in a cloud-basedcomputing environment rather than being tied to one or more specificphysical network computers.

Further, network computer 300 may also comprise hardware security module(HSM) 360 for providing additional tamper resistant safeguards forgenerating, storing or using security/cryptographic information such as,keys, digital certificates, passwords, passphrases, two-factorauthentication information, or the like. In some embodiments, hardwaresecurity module may be employ to support one or more standard public keyinfrastructures (PKI), and may be employed to generate, manage, or storekeys pairs, or the like. In some embodiments, HSM 360 may be astand-alone network computer, in other cases, HSM 360 may be arranged asa hardware card that may be installed in a network computer.

Additionally, in one or more embodiments (not shown in the figures),network computer 300 may include an embedded logic hardware deviceinstead of a CPU, such as, an Application Specific Integrated Circuit(ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic(PAL), or the like, or combination thereof. The embedded logic hardwaredevice may directly execute its embedded logic to perform actions. Also,in one or more embodiments (not shown in the figures), the networkcomputer may include one or more hardware microcontrollers instead of aCPU. In one or more embodiment, the one or more microcontrollers maydirectly execute their own embedded logic to perform actions and accesstheir own internal memory and their own external Input and OutputInterfaces (e.g., hardware pins or wireless transceivers) to performactions, such as System On a Chip (SOC), or the like.

Illustrative Logical System Architecture

FIG. 4 illustrates a logical schematic of system 400 for automatedtraining and selection of models for document analysis in accordancewith one or more of the various embodiments. In one or more of thevarious embodiments, one or more users may employ a client computer,such as, client computer 402 to provide a set of document attributes andone or more questions, such as, document attributes 404 or question 406to a machine learning engine, or the like, running on a documentmanagement server computer, such as, network computer 408. Accordingly,in some embodiments, the machine learning engine may be arranged toselect one or more document models, such as, document models 410 fortraining to answer question 406 based on attributes 404 and a corpus ofother documents.

In one or more of the various embodiments, system 400 may be arranged toanswer various kinds of questions about a document or its attributes. Insome embodiments, document attributes 404 may include one or more thevalues of known attributes and question 406 may be asking for system 400to provide a predicted value for an unavailable or unknown attribute. Insome embodiments, question 406 may include a request for system 400 todetermine if some or all of the values for attributes 404 may beanomalous. Also, in some embodiments, question 406 may include a requestto provide other or additional statistical evaluations or comparisons ofattributes 404 with similar attributes for other documents, such as,clustering, anomaly identification, or the like, or combination thereof.

In one or more of the various embodiments, system 400 may be arranged totrain one or more machine learning models, such as, document models 410to answer queries, such as, question 406 to provide insights 412associated with one or more working documents. In one or more of thevarious embodiments, queries or questions may include providing one ormore known values for one or more document attributes and one or moredocument attributes with unknown values. For example. in someembodiments, system 400 may provide a predicted value for the one ormore documents attributes without values. In some embodiments, documentinsights 412 may include one or more predicted values for one or moredocument attributes to provide users insight into their businessprocesses or operations based on the predictions.

For example, in some embodiments, contract document may have severalattributes that may provide insight into how long it may take to executeor finalize the agreement between the contracting parties. Likewise, insome embodiments, insights may include: risk assessments, identifyingsimilar documents (e.g., clustering), identifying anomalous documents,or the like, or combination thereof.

In some embodiments, system 400 may be arranged to selectively train oneor more candidate ML models to answer the same query (e.g., question406). Accordingly, in some embodiments, the ML models may be compared orotherwise ranked based on an accuracy score associated with theirrespective predictions. In some embodiments, the particular trainingmethod or scoring method may vary depending on the ML model.

In one or more of the various embodiments, the different ML models mayinclude models arranged to use different training techniques orstatistical techniques, including, linear regression, lasso regression,ridge regression, decision tree, random forest, logistic regression, orthe like, or combination thereof. Further, in some embodiments, variousheuristic methods or processes may be employed as well. In someembodiments, a system, such as, system 400 may be arranged to employconfiguration information, such as, rule based policies, pre-definedfeature weights, pattern matching, scripts (e.g., computer readableinstructions), or the like, to select candidate ML model to train toanswer a given query for a set of document attributes. In someembodiments, this configuration information may be provided from varioussources, including, configuration files, databases, user input, built-indefaults, or the like, or combination thereof.

In one or more of the various embodiments, the trained candidate MLmodels may be further evaluated to rank them against each other byrunning them against a set of validation documents. Accordingly, in someembodiments, one or more of the top ranked ML models may be selected toprovide an answer in response to questions, such as, question 406 basedon document attributes, such as, document attributes 404.

FIG. 5 illustrates a logical schematic of system 500 for automatedtraining and select of models for document or contract analysis inaccordance with one or more of the various embodiments. As describedabove, in some embodiments, an organization may employ one or morecomputers, such as, computer 502, to provide a corpus of documents to adocument analysis server computer, such as, document analysis servercomputer 506. Accordingly, in one or more of the various embodiments,one or more of a document engine, insight engine, machine learningengine, or the like, may be arranged to pre-process the documents toidentify attributes or features or the documents that may be suitablefor using, such as, document attributes 508, or the like.

Also, in one or more of the various embodiments, some or all or theoriginal documents (e.g., documents 504) may be divided into validationdocuments and training documents. For example, in some embodiments, thedocument engine, or the like, may be arranged to randomly select 25% ofthe document corpus to be validation document set with the 75% remainderbeing used as the training document set. Thus, in some embodiments, theML models may be trained using a training set, such as, trainingdocuments 510. And, in some embodiments, a validation set, such as,validation documents 512 may be used to evaluate and rank the one ormore candidate ML models.

In one or more of the various embodiments, document models 514represents the models used for answering queries or questions. In someembodiments, models 514 may include, one or more machine learningsmodels, one or more model templates, one or more partiallytrained/configured models, or the like.

In some embodiments, the document corpus needs to be pre-processed forthe purpose of analysis. In some embodiments, the pre-processing mayinclude one or more data mining techniques that may resolves issues,such as, incomplete or inconsistent data. In some embodiments, thedocument engine may be arranged to determine if a document attribute maybe numeric or categorical. In some embodiments, a numerical attributemay be an attribute where the measurement or number has a numericalmeaning. For example, “total contract value”, “number of cycle days” maybe examples of numerical attributes. In contrast, categorical attributesmay take on values that may be names or labels. For example, “country”,“state” are examples of categorical attributes.

Further, in one or more of the various embodiments, the document enginemay be arranged to determine if an attribute has valid data for thepurpose of analysis. Accordingly, in some embodiments, the documentengine may be arranged to execute one or more sanity check operations todetermine if an attribute type or feature is suitable for analysis. Forexample, if the data for a categorical attribute contains more than 90%unique values or a particular value occurs more than 90% of the totalcount, the attribute might not be considered suitable for the purpose ofanalysis. Similarly, in some embodiments, for a numerical attribute, ifa particular value occurs more than 90% of the total count, theattribute might not be considered suitable for analysis. Also, in someembodiments, records with missing values in dependent variables(attributes for which prediction is needed) may be discarded.Alternatively, in some embodiments, some records may have values missingfor certain attributes. In some embodiments, the document engine may bearranged to employ one or more statistical methods, such as, mean, mode,or the like, for numeric values attributes or most common values forcategorically valued attributes to impute missing values.

Generalized Operations

FIGS. 6-9 represent generalized operations for automated training andselection of models for document or contract analysis in accordance withone or more of the various embodiments. In one or more of the variousembodiments, processes 600, 700, 800, and 900 described in conjunctionwith FIGS. 6-9 may be implemented by or executed by one or moreprocessors on a single network computer (or network monitoringcomputer), such as network computer 300 of FIG. 3. In other embodiments,these processes, or portions thereof, may be implemented by or executedon a plurality of network computers, such as network computer 300 ofFIG. 3. In yet other embodiments, these processes, or portions thereof,may be implemented by or executed on one or more virtualized computers,such as, those in a cloud-based environment. However, embodiments arenot so limited and various combinations of network computers, clientcomputers, or the like may be utilized. Further, in one or more of thevarious embodiments, the processes described in conjunction with FIGS.6-9 may be used automated training and selection of models for documentor contract analysis in accordance with at least one of the variousembodiments or architectures such as those described in conjunction withFIGS. 4-5. Further, in one or more of the various embodiments, some orall of the actions performed by processes 600, 700, 800, and 900 may beexecuted in part by document engine 322, insight engine 324, or machinelearning engine 326 running on one or more processors of one or morenetwork computers.

FIG. 6 illustrates an overview flowchart of process 600 for automatedtraining and select of models for document or contract analysis inaccordance with one or more of the various embodiments. After a startblock, at block 602, in one or more of the various embodiments, variousdocuments may be provided to a document engine. As described above, anorganization may provide documents, including pending contracts,impending contracts, historical contracts, archived contracts, or thelike, as well as various related or supporting documents, including,work orders, invoices, letters of intent, job/work descriptions, otherdocuments, or the like.

In one or more of the various embodiments, the organization may providedigital copies of some or all of the documents. For example, some or allof the documents may be provided via digital media. Alternatively, orconcurrently, in some embodiments, the organization may share access tosome or all of the documents with the document engine. For example, insome embodiments, an organization may provide a document engine accessto document management tools that enable the document engine to accessthe documents.

At block 604, in one or more of the various embodiments, the documentengine may be arranged to analyze the documents to identify one or moredocument attributes that may be suitable for training models that may besuitable for answering user queries directed to new or existingdocuments.

At block 606, in one or more of the various embodiments, the documentengine may be arranged to divide the documents into training documentsand validation documents. For example, in some embodiments, the documentengine may be arranged to randomly assign 75% of the documents to thetraining set and assign the 25% remainder to a validation set.

In some embodiments, document engine may be arranged to execute one ormore rules or heuristics that consider one or more attributes orcharacteristics of a document to determine whether to assign it to thetraining set or a validation set. Likewise, in some embodiments, thedocument engine may be arranged to execute rules or heuristics that mayexclude one or more documents from either set based on one or morecharacteristics of the excluded documents, such as, document age,document subject matter, document type, document author, contractparties, or the like. For example, in some embodiments, the documentengine may be arranged to exclude old documents associated with aparticular country that has under gone significant social, political, oreconomic disruption because the disruptions may cause one or more of theinsights determined from the old contracts misleading. In someembodiments, a document engine may be arranged to employ configurationinformation, such as, rule based policies, pattern matching, scripts(e.g., computer readable instructions), or the like, to determine if adocument may be assigned to a training set or a validation set. In someembodiments, this configuration information may be provided from varioussources, including, configuration files, databases, user input, built-indefaults, or the like, or combination thereof.

At block 608, in one or more of the various embodiments, the documentengine may receive a document query from a user. In some embodiments,the query may be provided to an insight engine, machine learning engine,or document engine, or the like. In some embodiments, the query mayinclude one or more document attributes associated with a workingdocument that may be under analysis by a user. In some embodiments, thequery may include one or more document attributes of the workingdocument that have known values and one or more document attributes fromthe working document that have unknown values. For example, a query mayinclude five attributes with that have known values, such as,(A,_,C,D,E,F) to predict an unknown value for B.

At block 610, in one or more of the various embodiments, a machinelearning engine may be arranged to train two or more machine learningmodels based on the query and the training documents. In one or more ofthe various embodiments, the machine learning engine may select two ormore machine learning models from a pool of machine learning models. Insome embodiments, one or more of the machine learning models may bepartially or fully trained (pre-trained). Accordingly, in someembodiments, if the current query is a duplicate of a previous querythere may be one or more previously trained machine learning models thatmay be available.

In one or more of the various embodiments, the machine learning enginemay be arranged to select the one or more machine learning models fromthe pool of available machine learning models based on configurationinformation.

At block 612, in one or more of the various embodiments, the machinelearning engine may be arranged to evaluate the trained machine learningmodels based on the query and the validation documents. In one or moreof the various embodiments, the machine learning engine may be arrangedto assign each trained machine learning model an accuracy score. Theaccuracy may be a normalized value that enables different types ofmachine learning models to be compared to each other. Accordingly, inone or more of the various embodiments, each type of machine learningmodel may be associated with an accuracy function that may be directedto that particular type of machine learning model. Further, in someembodiments, user preferences or other configuration information may beemployed to modify or weight the accuracy score for one or more machinelearning models. Accordingly, in some embodiments, users or operatorsmay bias the accuracy score of one or more favored machine learningmodels.

At block 614, in one or more of the various embodiments, the machinelearning engine may be arranged to rank the trained machine learningmodels based on their associated accuracy score. In one or more of thevarious embodiments, the machine learning engine may be arranged toassociate an accuracy score with the machine learning models based onhow successful they were when responding to the query. Accordingly, insome embodiments, the machine learning models may be ranked based ontheir respective accuracy scores.

At block 616, in one or more of the various embodiments, the machinelearning engine may be arranged to provide a response to the query basedon the query and one or more of the top ranked machine learning models.Next, control may be returned to a calling process.

FIG. 7 illustrates a flowchart of process 700 for determining documentattributes for automated training and select of models for document orcontract analysis in accordance with one or more of the variousembodiments. After a start block, at block 702, in one or more of thevarious embodiments, a corpus of documents may be provided to documentengine. In some embodiments, an organization may provide some or all ofits documents to a document analysis platform, such as, documentanalysis server computer 116. In some embodiments, the organization mayprovide documents of one or more types, such as, contracts, licenseagreements, or the like.

As described above, documents may be provided in raw/original form orthey may be provided after some amount of pre-processing. For example,in some embodiments, documents may be provided via integration with adocument management system or contract management system that includesor maintains meta-data that may be relevant for determining one or moredocument attributes.

At block 704, in one or more of the various embodiments, the documentengine may be arranged to data mine the provided documents to identifyone or more candidate document attributes. In some embodiments,attributes may be considered features of the documents or its contents.In some embodiments, the available types of document attributes may bepre-defined based on the type document. In some embodiments, thedocument engine may be arranged to recognize one or more common documentattributes, such as, author, parties, identification of various parts orclauses of documents, or the like. Also, in some embodiments, custom orattributes otherwise unique to an organization or its it documents maybe defined as well. Accordingly, in one or more of the variousembodiments, the document engine may be arranged to employ configurationinformation that defines one or more rules, conditions, patternmatching, or the like, that may be executed by the document engine toidentify one or more candidate document attributes.

At block 706, in one or more of the various embodiments, the documentengine may be arranged to evaluate the one or more candidate documentattributes. As mentioned above, the document engine may be arranged toexecute one or more heuristics, rules, computer readable instructions,or the like, to evaluate whether one or more document attributes aresuitable for including in the modeling. In one or more of the variousembodiments, one or more general or specific criteria may be evaluatedto determine if a candidate document attribute is suitable for includingin the modeling process. For example, in some embodiments, if acategorical attribute contains more than 90% unique values or aparticular value occurs more than 90% of the total count, the attributemay not be considered for the purpose of analysis because it would notbe useful for making predictions about the values of other documentattributes.

At block 708, in one or more of the various embodiments, the documentengine may be arranged to determine one or more viable documentattributes based the evaluation of the candidate document attributes.Candidate document attributes that may be determined as suitable formodeling may be identified and made available as document attributes.

In one or more of the various embodiments, these attributes may be madeavailable to other applications or user interfaces that users may beemploy to form queries or questions to predict other documentattributes. Next, control may be returned to calling process.

FIG. 8 illustrates a flowchart of process 800 for predicting unknowndocument attributes of a working document in accordance with one or moreof the various embodiments. After a start block, at block 802, in one ormore of the various embodiments, one or more working documents may beprovided to the document engine. In some embodiments, a working documentmay be a document, such as, a contract, license, or the like, that auser may be researching or processing. For example, the working documentmay be a proposed contract presented during the negotiation of a dealbetween one or more parties. In some embodiments, working document maybe finalized or otherwise in-force. Accordingly, in some embodiments,the working document may be a document of a known type that is beingdeveloped (e.g., negotiated) or one that has already been finalized.

In one or more of the various embodiments, the working document may berepresented by a set of one or more document attributes rather than anactual document. For example, in some embodiments, a user performing awhat-if analysis may provide a set of known or proposed documentattributes rather than an actual document.

At block 804, in one or more of the various embodiments, the documentengine may be arranged to determine one or more known document attributevalues for the one or more working documents. In one or more of thevarious embodiments, the document engine may be arranged to scan orparse the working document to automatically discover one or more valuesfor one or more document attributes of the working document. In someembodiments, the working document may be associated with meta-data thatdefines the relevant document attributes and the known values, if any.In some embodiments, the document engine may receive one or moredocument attributes or document attribute values from a user that may beanalyzing the working document. In some embodiments, the document enginemay identify one or more document attributes that are missing or havemissing values.

At block 806, in one or more of the various embodiments, one or moredocument attributes of the working document that are missing values maybe determined or identified. In some embodiments, the document enginemay generate a user interface that enables one or more users to identifyone or more unknown valued document attributes. Also, in one or more ofthe various embodiments, the document engine may be arranged toautomatically determine one or more document attributes having unknownvalues. For example, in one or more of the various embodiments, adocument engine may employ configuration information to determine one ormore documents attributes with unknown values.

In one or more of the various embodiments, one or more of the determinedunknown valued document attributes and the one or more known-valueddocument attributes may be included in a query that may be provided to amachine learning engine.

At block 808, in one or more of the various embodiments, the machinelearning engine may be arranged to employ the training documents totrain one or more machine learning models to predict values for the oneor more document attribute having unknown values. As described above, inone or more of the various embodiments, there may be data store of oneor more untrained machine learning models that represent variousmodeling or machine learning techniques.

Accordingly, in one or more of the various embodiments, the machinelearning engine may be arranged to select two or more machine learningmodels and train them to predict values for the one or more unknownvalued document attributes. In some embodiments, the machine learningengine may be arranged to select one or more particular untrainedmachine learning models based on a document type of the workingdocument.

Also, in some embodiments, the machine learning engine may selectuntrained machine learning models based on the query inputs. In someembodiments, one or more document attributes or attribute sets may beassociated with particular untrained machine learning models. Forexample, in some embodiments, categorically valued document attributesmay be associated with different machine learning models than numericalvalued or continuously valued document attributes.

In some embodiments, one or more particular document attributes ordocument attribute sets may be associated with definite known values. Insuch cases, the corresponding unknown values may be provided based onheuristics, lookups, well-known formulas, or the like, rather thanrequiring the use of machine learning models or the training thereof.Accordingly, in some embodiments, information regarding this directmapping/computation of attribute values may be provided viaconfiguration information. In some embodiments, some or all of theconfiguration information may be provided by various sources, includingconfiguration files, configuration databases/registries, rule-basedpolicies, computer readable instructions, built-in values, defaultvalues, user input, or the like, or combination thereof.

Also, in one or more of the various embodiments, one or more machinelearning models may be fully trained or partially trained based onprevious queries. Accordingly, in some embodiments, the one or moretrained or partially trained machine learning model may be stored andindexed based on keys generated based on document types, question/querytype (e.g., missing value prediction, clustering, anomaly detection, orthe like), valued document attributes, unknown valued documentattributes, or the like. Accordingly, in some embodiments, thepre-trained machine learning models may be included in the trainingprocess. Alternatively, in some embodiments, one or more of thepre-trained machine learning models may be omitted from training.

In one or more of the various embodiments, the machine learning enginemay be arranged to perform one or more actions associated with themachine learning techniques associated with a given machine learningmodel. In some embodiments, there may be one or more other actions,including, customized processing, filters, heuristics, normalizations,or the like, that may be associated with one or more machine learningmodels. In some embodiments, one or more of these actions may beincluded in the training process for one or more machine learningmodels. Accordingly, in some embodiments, the machine learning enginemay determine one or more training actions for given machine learningmodel based on configuration information.

At block 810, in one or more of the various embodiments, the machinelearning engine may be arranged to evaluate the accuracy of the trainedmachine learning models using the validation documents. In one or moreof the various embodiments, after the machine learning models have beentrained using the training documents, the trained machine learningmodels may be tested using the validation documents. Each model may bescored based on how well it predicts the missing attributes or otherwiseanswers the query. For example, in some embodiments, an accuracy scoremay be computed based on one or more formulas that provide normalizedscoring system. In some embodiments, the scoring formulas may bemodified or otherwise tuned based on the type of machine learning modelbeing evaluated.

At block 812, in one or more of the various embodiments, the documentengine may be arranged to employ the high scoring machine learningmodels to predict one or more values for the unknown documentattributes. In some embodiments, the document engine may rank themachine learning models based on their accuracy score. Accordingly, inone or more of the various embodiments, the document engine may selectone or more of the top ranked machine learning models for use inanswering the supplied query. For example, the highest ranked machinelearning model may be used to predict the values for the one or moreunknown valued document attributes of the working document. Next,control may be returned to a calling process.

FIG. 9 illustrates a flowchart of process 900 for training machinelearning models used for automated training and select of models fordocument or contract analysis in accordance with one or more of thevarious embodiments. After a start block, at block 902, in one or moreof the various embodiments, a machine learning engine may be providedtraining data, such as, training documents that may be selected from acorpus of documents provided by an organization. In one or more of thevarious embodiments, a machine learning engine may be provided trainingdata in the form of a large collection of documents of a known qualityand a known document type.

In some embodiments, the training data may have previously beenclassified or evaluated. In one or more of the various embodiments, thetraining data may have been preprocessed by a document engine or machinelearning engine as described above. Accordingly, in one or more of thevarious embodiments, the training data may include labels, tags,meta-data, document quality scores, or the like.

Accordingly, the documents in the training data may be associated withmeta-data generated by the document engine during preprocessing. In someembodiments, the meta-data may include label information or documentattribute information that may be used by the machine learning enginefor training one or more machine learning (ML) models.

At block 904, in one or more of the various embodiments, the machinelearning engine may be arranged to train one or more ML models using thetraining data. In some embodiments, training a machine learning model isthe process of iteratively improving the machine learning modelprediction results by looping or otherwise evaluating documents in thetraining set. In some embodiments, depending on the particular machinelearning model, machine learning engines may be arranged to update oneor more weights or bias values included in the machine learning model.In some embodiments, training may be complete if an acceptable errorthreshold is reached, or if subsequent training iterations fail toimprove the accuracy of the machine learning model.

In one or more of the various embodiments, the training methods maydepend on the type of ML model being trained. Accordingly, in someembodiments, the machine learning engine may be arranged to performdifferent training actions for different machine learning models. Theparticular training process or training instructions may be provided viaconfiguration information. In some embodiments, some or all of theconfiguration information may be provided by various sources, includingconfiguration files, configuration databases/registries, rule-basedpolicies, computer readable instructions, built-in values, defaultvalues, user input, or the like, or combination thereof.

At block 906, in one or more of the various embodiments, the documentengine or machine learning engine may be arranged to evaluate thetrained machine learning model. In one or more of the variousembodiments, a portion of the validation documents that may have knowncharacteristics or document attributes may be evaluated using themachine learning models. Accordingly, in one or more of the variousembodiments, machine learning models that predict the known documentattributes of validation documents with an accuracy rate that exceeds adefined threshold value may be considered sufficiently trained. Note, insome embodiments, different machine learning models may be associatedwith different accuracy threshold values. For example, some machinelearning model types may be intended for gross classification that doesnot require precise accuracy. In some embodiments, other machinelearning models that may require increased accuracy or precision may beassociated with threshold values that correspond to increased accuracyor precision.

At decision block 908, in one or more of the various embodiments, if thetrained ML models are sufficiently trained, control may flow to block912; otherwise, control may flow to block 910. In one or more of thevarious embodiments, a machine learning engine may be arranged todetermine if a machine learning model is trained based on comparing oneor more performance metrics associated with the machine learning model.The particular performance metrics may vary depending on the type ofmachine learning model. In some embodiments, the number of availableinputs or the type of answer/response being sought may also influencethe criteria or performance metrics for a given machine learning model.Accordingly, in one or more of the various embodiments, configurationinformation or meta-data associated with a machine learning model mayinclude rules, conditions, threshold values, or the like, that may beemployed by the machine learning engine to determine if a machinelearning model is sufficiently trained.

At block 910, in one or more of the various embodiments, the one or moreof the ML models or one or more of the training routines may bemodified. In one or more of the various embodiments, the machinelearning engine may be arranged to automatically modify one or moreparameters of the one or more ML models that require retraining. Inother embodiments, the machine learning engine may enable datascientists to modify the ML models or select different ML models. Next,in some embodiments, control may loop back to block 904, to re-train theML models.

At block 912, in one or more of the various embodiments, the machinelearning engine may be arranged to ranks the trained machine learningmodel based on their accuracy scores. At block 914, in one or more ofthe various embodiments, the machine learning engine may be arranged toprovide the top ranked machine learning model for answering a queryprovided by a user. Next, control may be returned to a calling process.

It will be understood that each block of the flowchart illustration, andcombinations of blocks in the flowchart illustration, can be implementedby computer program instructions. These program instructions may beprovided to a processor to produce a machine, such that theinstructions, which execute on the processor, create means forimplementing the actions specified in the flowchart block or blocks. Thecomputer program instructions may be executed by a processor to cause aseries of operational steps to be performed by the processor to producea computer-implemented process such that the instructions, which executeon the processor to provide steps for implementing the actions specifiedin the flowchart block or blocks. The computer program instructions mayalso cause at least some of the operational steps shown in the blocks ofthe flowchart to be performed in parallel. Moreover, some of the stepsmay also be performed across more than one processor, such as mightarise in a multi-processor computer system. In addition, one or moreblocks or combinations of blocks in the flowchart illustration may alsobe performed concurrently with other blocks or combinations of blocks,or even in a different sequence than illustrated without departing fromthe scope or spirit of the invention.

Accordingly, blocks of the flowchart illustration support combinationsof means for performing the specified actions, combinations of steps forperforming the specified actions and program instruction means forperforming the specified actions. It will also be understood that eachblock of the flowchart illustration, and combinations of blocks in theflowchart illustration, can be implemented by special purpose hardwarebased systems, which perform the specified actions or steps, orcombinations of special purpose hardware and computer instructions. Theforegoing example should not be construed as limiting or exhaustive, butrather, an illustrative use case to show an implementation of at leastone of the various embodiments of the invention.

Further, in one or more embodiments (not shown in the figures), thelogic in the illustrative flowcharts may be executed using an embeddedlogic hardware device instead of a CPU, such as, an Application SpecificIntegrated Circuit (ASIC), Field Programmable Gate Array (FPGA),Programmable Array Logic (PAL), or the like, or combination thereof. Theembedded logic hardware device may directly execute its embedded logicto perform actions. In one or more embodiment, a microcontroller may bearranged to directly execute its own embedded logic to perform actionsand access its own internal memory and its own external Input and OutputInterfaces (e.g., hardware pins or wireless transceivers) to performactions, such as System On a Chip (SOC), or the like.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A method for managing contract documents over anetwork using one or more processors that execute instructions toperform actions, comprising: instantiating a machine learning (ML)engine to perform actions, including: determining one or more oftraining contract documents and one or more validation contractdocuments that are randomly selected from a plurality of contractdocuments; employing two or more evaluators to determine one or moreattributes associated with the plurality of contract documents, whereinthe one or more attributes are associated with one or more localizationfeatures that are associated with the plurality of contract documents,and wherein the two or more evaluators include a semantic evaluator anda textual evaluator that employ the one or more localization features tovalidate one or more of categories of contract clauses or contractdocument types, and wherein geolocation information is employed toselect the one or more localization features, including time zones,languages, currencies, currency formatting, or calendar formatting; andin response to a request to predict one or more attribute values of aselected contract document, performing further actions, including:training a plurality of ML models to predict the one or more attributevalues based on the one or more training contract documents and the oneor more attributes, wherein each trained ML model is associated with atraining accuracy score; determining one or more candidate ML modelsfrom the plurality of trained ML models based on each associatedtraining accuracy score exceeding a threshold value; evaluating the oneor more candidate ML models based on the request and the one or morevalidation contract documents, wherein each of the one or more evaluatedcandidate ML models is ranked; and generating one or more confirmed MLmodels based on the one or more ranked candidate ML models, wherein theone or more confirmed ML models are employed to answer the request andpredict the one or more attribute values of the selected contractdocument, wherein employing the one or more confirmed ML models improvesboth efficiency of employed computing resources and accuracy of theanswer to the request.
 2. The method of claim 1, wherein one or more ofthe plurality of ML models were either fully trained or partiallytrained prior to receiving the request.
 3. The method of claim 1,wherein determining the one or more candidate ML models, furthercomprises, in response to the request matching one or more previousrequests, modifying the one or more candidate ML models to include oneor more confirmed ML models that were previously used for answering theone or more matched requests.
 4. The method of claim 1, whereindetermining the one or more attributes, further comprises: determiningone or more categorical attributes that have values that are names orlabels; and determining one or more numerical attributes that havevalues that represent a numerical meaning.
 5. The method of claim 1,wherein the one or more attributes, include one or more of value, numberof cycle dates, geographic location, subject matter, duration, deliverydate, entities, forum, or venue.
 6. The method of claim 1, wherein theML engine performs further actions, including: providing one or morecontract document types based on a selection of attributes or attributevalues, wherein each contract document is associated with the one ormore contract document types based on the one or more attributesincluded in the contract document; and associating one or more of theplurality of ML models with the one or more contract document types. 7.The method of claim 1, wherein the ML engine performs further actions,including, executing the one or more confirmed ML models to identify theone or more attributes of the contract document that are outliers. 8.The method of claim 1, wherein the ML engine performs further actions,including, executing the one or more confirmed ML models to identify oneor more contract document clusters based on the one or more attributes,wherein the plurality of contract documents are associated with the oneor more clusters based on values of the one or more attributes that areassociated with each cluster of contract documents.
 9. A system formanaging contract documents comprising: a network computer, comprising:a transceiver that communicates over the network; a memory that storesat least instructions; and one or more processors that executeinstructions that perform actions, including: instantiating a machinelearning (ML) engine to perform actions, including: determining one ormore of training contract documents and one or more validation contractdocuments that are randomly selected from a plurality of contractdocuments; employing two or more evaluators to determine one or moreattributes associated with the plurality of contract documents, whereinthe one or more attributes are associated with one or more localizationfeatures that are associated with the plurality of contract documents,and wherein the two or more evaluators include a semantic evaluator anda textual evaluator that employ the one or more localization features tovalidate one or more of categories of contract clauses or contractdocument types, and wherein geolocation information is employed toselect the one or more localization features, including time zones,languages, currencies, currency formatting, or calendar formatting; andin response to a request to predict one or more attribute values of aselected contract document, performing further actions, including:training a plurality of ML models to predict the one or more attributevalues based on the one or more training contract documents and the oneor more attributes, wherein each trained ML model is associated with atraining accuracy score; determining one or more candidate ML modelsfrom the plurality of trained ML models based on each associatedtraining accuracy score exceeding a threshold value; evaluating the oneor more candidate ML models based on the request and the one or morevalidation contract documents, wherein each of the one or more evaluatedcandidate ML models is ranked; and generating one or more confirmed MLmodels based on the one or more ranked candidate ML models, wherein theone or more confirmed ML models are employed to answer the request andpredict the one or more attribute values of the selected contractdocument, wherein employing the one or more confirmed ML models improvesboth efficiency of employed computing resources and accuracy of theanswer to the request; and a client computer, comprising: a transceiverthat communicates over the network; a memory that stores at leastinstructions; and one or more processors that execute instructions thatperform actions, including: providing one or more of the plurality ofcontract documents or the request.
 10. The system of claim 9, whereinone or more of the plurality of ML models were either fully trained orpartially trained prior to receiving the request.
 11. The system ofclaim 9, wherein determining the one or more candidate ML models,further comprises, in response to the request matching one or moreprevious requests, modifying the one or more candidate ML models toinclude one or more confirmed ML models that were previously used foranswering the one or more matched requests.
 12. The system of claim 9,wherein determining the one or more attributes, further comprises:determining one or more categorical attributes that have values that arenames or labels; and determining one or more numerical attributes thathave values that represent a numerical meaning.
 13. The system of claim9, wherein the one or more attributes, include one or more of value,number of cycle dates, geographic location, subject matter, duration,delivery date, entities, forum, or venue.
 14. The system of claim 9,wherein the ML engine performs further actions, including: providing oneor more contract document types based on a selection of attributes orattribute values, wherein each contract document is associated with theone or more contract document types based on the one or more attributesincluded in the contract document; and associating one or more of theplurality of ML models with the one or more contract document types. 15.The system of claim 9, wherein the ML engine performs further actions,including, executing the one or more confirmed ML models to identify theone or more attributes of the contract document that are outliers. 16.The system of claim 9, wherein the ML engine performs further actions,including, executing the one or more confirmed ML models to identify oneor more contract document clusters based on the one or more attributes,wherein the plurality of contract documents are associated with the oneor more clusters based on values of the one or more attributes that areassociated with each cluster of contract documents.
 17. A processorreadable non-transitory storage media that includes instructions formanaging contract documents over a network, wherein execution of theinstructions by one or more processors on one or more network computersperforms actions, comprising: instantiating a machine learning (ML)engine to perform actions, including: determining one or more oftraining contract documents and one or more validation contractdocuments that are randomly selected from a plurality of contractdocuments; employing two or more evaluators to determine one or moreattributes associated with the plurality of contract documents, whereinthe one or more attributes are associated with one or more localizationfeatures that are associated with the plurality of contract documents,and wherein the two or more evaluators include a semantic evaluator anda textual evaluator that employ the one or more localization features tovalidate one or more of categories of contract clauses or contractdocument types, and wherein geolocation information is employed toselect the one or more localization features, including time zones,languages, currencies, currency formatting, or calendar formatting; andin response to a request to predict one or more attribute values of aselected contract document, performing further actions, including:training a plurality of ML models to predict the one or more attributevalues based on the one or more training contract documents and the oneor more attributes, wherein each trained ML model is associated with atraining accuracy score; determining one or more candidate ML modelsfrom the plurality of trained ML models based on each associatedtraining accuracy score exceeding a threshold value; evaluating the oneor more candidate ML models based on the request and the one or morevalidation contract documents, wherein each of the one or more evaluatedcandidate ML models is ranked; and generating one or more confirmed MLmodels based on the one or more ranked candidate ML models, wherein theone or more confirmed ML models are employed to answer the request andpredict the one or more attribute values of the selected contractdocument, wherein employing the one or more confirmed ML models improvesboth efficiency of employed computing resources and accuracy of theanswer to the request.
 18. The media of claim 17, wherein one or more ofthe plurality of ML models were either fully trained or partiallytrained prior to receiving the request.
 19. The media of claim 17,wherein determining the one or more candidate ML models, furthercomprises, in response to the request matching one or more previousrequests, modifying the one or more candidate ML models to include oneor more confirmed ML models that were previously used for answering theone or more matched requests.
 20. The media of claim 17, whereindetermining the one or more attributes, further comprises: determiningone or more categorical attributes that have values that are names orlabels; and determining one or more numerical attributes that havevalues that represent a numerical meaning.
 21. The media of claim 17,wherein the one or more attributes, include one or more of value, numberof cycle dates, geographic location, subject matter, duration, deliverydate, entities, forum, or venue.
 22. The media of claim 17, wherein theML engine performs further actions, including: providing one or morecontract document types based on a selection of attributes or attributevalues, wherein each contract document is associated with the one ormore contract document types based on the one or more attributesincluded in the contract document; and associating one or more of theplurality of ML models with the one or more document types.
 23. Themedia of claim 17, wherein the ML engine performs further actions,including, executing the one or more confirmed ML models to identify theone or more attributes of the contract document that are outliers.
 24. Anetwork computer for managing contract documents, comprising: atransceiver that communicates over the network; a memory that stores atleast instructions; and one or more processors that execute instructionsthat perform actions, including: instantiating a machine learning (ML)engine to perform actions, including: determining one or more oftraining contract documents and one or more validation contractdocuments that are randomly selected from a plurality of contractdocuments; employing two or more evaluators to determine one or moreattributes associated with the plurality of contract documents, whereinthe one or more attributes are associated with one or more localizationfeatures that are associated with the plurality of contract documents,and wherein the two or more evaluators include a semantic evaluator anda textual evaluator that employ the one or more localization features tovalidate one or more of categories of contract clauses or contractdocument types, and wherein geolocation information is employed toselect the one or more localization features, including time zones,languages, currencies, currency formatting, or calendar formatting; andin response to a request to predict one or more attribute values of aselected contract document, performing further actions, including:training a plurality of ML models to predict the one or more attributevalues based on the one or more training contract documents and the oneor more attributes, wherein each trained ML model is associated with atraining accuracy score; determining one or more candidate ML modelsfrom the plurality of trained ML models based on each associatedtraining accuracy score exceeding a threshold value; evaluating the oneor more candidate ML models based on the request and the one or morevalidation contract documents, wherein each of the one or more evaluatedcandidate ML models is ranked; and generating one or more confirmed MLmodels based on the one or more ranked candidate ML models, wherein theone or more confirmed ML models are employed to answer the request andpredict the one or more attribute values of the selected contractdocument, wherein employing the one or more confirmed ML models improvesboth efficiency of employed computing resources and accuracy of theanswer to the request.
 25. The network computer of claim 24, wherein oneor more of the plurality of ML models were either fully trained orpartially trained prior to receiving the request.
 26. The networkcomputer of claim 24, wherein determining the one or more candidate MLmodels, further comprises, in response to the request matching one ormore previous requests, modifying the one or more candidate ML models toinclude one or more confirmed ML models that were previously used foranswering the one or more matched requests.
 27. The network computer ofclaim 24, wherein determining the one or more attributes, furthercomprises: determining one or more categorical attributes that havevalues that are names or labels; and determining one or more numericalattributes that have values that represent a numerical meaning.
 28. Thenetwork computer of claim 24, wherein the one or more attributes,include one or more of value, number of cycle dates, geographiclocation, subject matter, duration, delivery date, entities, forum, orvenue.
 29. The network computer of claim 24, wherein the ML engineperforms further actions, including: providing one or more contractdocument types based on a selection of attributes or attribute values,wherein each contract document is associated with the one or morecontract document types based on the one or more attributes included inthe contract document; and associating one or more of the plurality ofML models with the one or more contract document types.
 30. The networkcomputer of claim 24, wherein the ML engine performs further actions,including, executing the one or more confirmed ML models to identify oneor more contract document clusters based on the one or more attributes,wherein the plurality of contract documents are associated with the oneor more clusters based on values of the one or more attributes that areassociated with each cluster of contract documents.